{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b8a9d5c6-c939-42af-8aa0-3fa181807c72",
   "metadata": {},
   "source": [
    "# 实战：文档解析和分块处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f9d9b6-9011-45f1-b557-d6410e64bef7",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1.langchain document格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f7fb0962-e711-4f09-99ce-5418a7481732",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b32bd9a-87ce-45c6-94ad-815b84675703",
   "metadata": {},
   "outputs": [],
   "source": [
    "document = Document(page_content=\"hello, world\",\n",
    "                   metadata={\"source\": \"test meta data\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "40059281-472e-407a-a35b-4438077e2df0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello, world'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4b164bdb-a2c4-4ba5-bcff-c3aeaa7a61dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': 'test meta data'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1357ab89-7766-4362-8e8a-6f6a1cb84a9c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a45f4a86-e354-414a-b47d-1e9b602aadf4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ed84743a-728a-4c7b-816d-b0e4f05a1f6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "USER_AGENT environment variable not set, consider setting it to identify your requests.\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import WebBaseLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bef76f99-ed57-48ff-9190-70db85fc7209",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = WebBaseLoader(\"https://flask.palletsprojects.com/en/3.0.x/tutorial/layout/\")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c22d9f42-77f6-4f83-b660-a34005c4f580",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Project Layout — Flask Documentation (3.0.x)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Navigation\n",
      "\n",
      "\n",
      "index\n",
      "\n",
      "modules |\n",
      "\n",
      "next |\n",
      "\n",
      "previous |\n",
      "Flask Documentation (3.0.x) »\n",
      "Tutorial »\n",
      "Project Layout\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Project Layout¶\n",
      "Create a project directory and enter it:\n",
      "$ mkdir flask-tutorial\n",
      "$ cd flask-tutorial\n",
      "\n",
      "\n",
      "Then follow the installation instructions to set\n",
      "up a Python virtual environment and install Flask for your project.\n",
      "The tutorial will assume you’re working from the flask-tutorial\n",
      "directory from now on. The file names at the top of each code block are\n",
      "relative to this directory.\n",
      "\n",
      "A Flask application can be as simple as a single file.\n",
      "\n",
      "hello.py¶\n",
      "from flask import Flask\n",
      "\n",
      "app = Flask(__name__)\n",
      "\n",
      "\n",
      "@app.route('/')\n",
      "def hello():\n",
      "    return 'Hello, World!'\n",
      "\n",
      "\n",
      "\n",
      "However, as a project gets bigger, it becomes overwhelming to keep all\n",
      "the code in one file. Python projects use packages to organize code\n",
      "into multiple modules that can be imported where needed, and the\n",
      "tutorial will do this as well.\n",
      "The project directory will contain:\n",
      "\n",
      "flaskr/, a Python package containing your application code and\n",
      "files.\n",
      "tests/, a directory containing test modules.\n",
      ".venv/, a Python virtual environment where Flask and other\n",
      "dependencies are installed.\n",
      "Installation files telling Python how to install your project.\n",
      "Version control config, such as git. You should make a habit of\n",
      "using some type of version control for all your projects, no matter\n",
      "the size.\n",
      "Any other project files you might add in the future.\n",
      "\n",
      "By the end, your project layout will look like this:\n",
      "/home/user/Projects/flask-tutorial\n",
      "├── flaskr/\n",
      "│   ├── __init__.py\n",
      "│   ├── db.py\n",
      "│   ├── schema.sql\n",
      "│   ├── auth.py\n",
      "│   ├── blog.py\n",
      "│   ├── templates/\n",
      "│   │   ├── base.html\n",
      "│   │   ├── auth/\n",
      "│   │   │   ├── login.html\n",
      "│   │   │   └── register.html\n",
      "│   │   └── blog/\n",
      "│   │       ├── create.html\n",
      "│   │       ├── index.html\n",
      "│   │       └── update.html\n",
      "│   └── static/\n",
      "│       └── style.css\n",
      "├── tests/\n",
      "│   ├── conftest.py\n",
      "│   ├── data.sql\n",
      "│   ├── test_factory.py\n",
      "│   ├── test_db.py\n",
      "│   ├── test_auth.py\n",
      "│   └── test_blog.py\n",
      "├── .venv/\n",
      "├── pyproject.toml\n",
      "└── MANIFEST.in\n",
      "\n",
      "\n",
      "If you’re using version control, the following files that are generated\n",
      "while running your project should be ignored. There may be other files\n",
      "based on the editor you use. In general, ignore files that you didn’t\n",
      "write. For example, with git:\n",
      "\n",
      ".gitignore¶\n",
      ".venv/\n",
      "\n",
      "*.pyc\n",
      "__pycache__/\n",
      "\n",
      "instance/\n",
      "\n",
      ".pytest_cache/\n",
      ".coverage\n",
      "htmlcov/\n",
      "\n",
      "dist/\n",
      "build/\n",
      "*.egg-info/\n",
      "\n",
      "\n",
      "\n",
      "Continue to Application Setup.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Navigation\n",
      "\n",
      "Overview\n",
      "\n",
      "Tutorial\n",
      "\n",
      "Previous: Tutorial\n",
      "Next: Application Setup\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Quick search\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "    © Copyright 2010 Pallets.\n",
      "      Created using Sphinx 8.0.2.\n",
      "    \n",
      "\n",
      "\n",
      " {'source': 'https://flask.palletsprojects.com/en/3.0.x/tutorial/layout/', 'title': 'Project Layout — Flask Documentation (3.0.x)', 'language': 'en'}\n"
     ]
    }
   ],
   "source": [
    "for doc in docs:\n",
    "    print(doc.page_content, doc.metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "90104369-31bd-47f9-beb2-7fb581eaf783",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(metadata={'source': './data/test.html', 'title': 'Project Layout — Flask Documentation (3.0.x)'}, page_content=\"\\n\\n\\n\\n\\nProject Layout — Flask Documentation (3.0.x)\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nNavigation\\n\\n\\nindex\\n\\nmodules |\\n\\nnext |\\n\\nprevious |\\nFlask Documentation (3.0.x) »\\nTutorial »\\nProject Layout\\n\\n\\n\\n\\n\\n\\n\\nProject Layout¶\\nCreate a project directory and enter it:\\n$ mkdir flask-tutorial\\n$ cd flask-tutorial\\n\\n\\nThen follow the installation instructions to set\\nup a Python virtual environment and install Flask for your project.\\nThe tutorial will assume you’re working from the flask-tutorial\\ndirectory from now on. The file names at the top of each code block are\\nrelative to this directory.\\n\\nA Flask application can be as simple as a single file.\\n\\nhello.py¶\\nfrom flask import Flask\\n\\napp = Flask(__name__)\\n\\n\\n@app.route('/')\\ndef hello():\\n    return 'Hello, World!'\\n\\n\\n\\nHowever, as a project gets bigger, it becomes overwhelming to keep all\\nthe code in one file. Python projects use packages to organize code\\ninto multiple modules that can be imported where needed, and the\\ntutorial will do this as well.\\nThe project directory will contain:\\n\\nflaskr/, a Python package containing your application code and\\nfiles.\\ntests/, a directory containing test modules.\\n.venv/, a Python virtual environment where Flask and other\\ndependencies are installed.\\nInstallation files telling Python how to install your project.\\nVersion control config, such as git. You should make a habit of\\nusing some type of version control for all your projects, no matter\\nthe size.\\nAny other project files you might add in the future.\\n\\nBy the end, your project layout will look like this:\\n/home/user/Projects/flask-tutorial\\n├── flaskr/\\n│   ├── __init__.py\\n│   ├── db.py\\n│   ├── schema.sql\\n│   ├── auth.py\\n│   ├── blog.py\\n│   ├── templates/\\n│   │   ├── base.html\\n│   │   ├── auth/\\n│   │   │   ├── login.html\\n│   │   │   └── register.html\\n│   │   └── blog/\\n│   │       ├── create.html\\n│   │       ├── index.html\\n│   │       └── update.html\\n│   └── static/\\n│       └── style.css\\n├── tests/\\n│   ├── conftest.py\\n│   ├── data.sql\\n│   ├── test_factory.py\\n│   ├── test_db.py\\n│   ├── test_auth.py\\n│   └── test_blog.py\\n├── .venv/\\n├── pyproject.toml\\n└── MANIFEST.in\\n\\n\\nIf you’re using version control, the following files that are generated\\nwhile running your project should be ignored. There may be other files\\nbased on the editor you use. In general, ignore files that you didn’t\\nwrite. For example, with git:\\n\\n.gitignore¶\\n.venv/\\n\\n*.pyc\\n__pycache__/\\n\\ninstance/\\n\\n.pytest_cache/\\n.coverage\\nhtmlcov/\\n\\ndist/\\nbuild/\\n*.egg-info/\\n\\n\\n\\nContinue to Application Setup.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nNavigation\\n\\nOverview\\n\\nTutorial\\n\\nPrevious: Tutorial\\nNext: Application Setup\\n\\n\\n\\n\\n\\nQuick search\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n    © Copyright 2010 Pallets.\\n      Created using Sphinx 8.0.2.\\n    \\n\\n\\n\")]\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import BSHTMLLoader\n",
    "loader = BSHTMLLoader('./data/test.html')\n",
    "data = loader.load()\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "32b2ef65-8660-42fe-8593-63eb04bf496d",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Project Layout — Flask Documentation (3.0.x)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Navigation\n",
      "\n",
      "\n",
      "index\n",
      "\n",
      "modules |\n",
      "\n",
      "next |\n",
      "\n",
      "previous |\n",
      "Flask Documentation (3.0.x) »\n",
      "Tutorial »\n",
      "Project Layout\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Project Layout¶\n",
      "Create a project directory and enter it:\n",
      "$ mkdir flask-tutorial\n",
      "$ cd flask-tutorial\n",
      "\n",
      "\n",
      "Then follow the installation instructions to set\n",
      "up a Python virtual environment and install Flask for your project.\n",
      "The tutorial will assume you’re working from the flask-tutorial\n",
      "directory from now on. The file names at the top of each code block are\n",
      "relative to this directory.\n",
      "\n",
      "A Flask application can be as simple as a single file.\n",
      "\n",
      "hello.py¶\n",
      "from flask import Flask\n",
      "\n",
      "app = Flask(__name__)\n",
      "\n",
      "\n",
      "@app.route('/')\n",
      "def hello():\n",
      "    return 'Hello, World!'\n",
      "\n",
      "\n",
      "\n",
      "However, as a project gets bigger, it becomes overwhelming to keep all\n",
      "the code in one file. Python projects use packages to organize code\n",
      "into multiple modules that can be imported where needed, and the\n",
      "tutorial will do this as well.\n",
      "The project directory will contain:\n",
      "\n",
      "flaskr/, a Python package containing your application code and\n",
      "files.\n",
      "tests/, a directory containing test modules.\n",
      ".venv/, a Python virtual environment where Flask and other\n",
      "dependencies are installed.\n",
      "Installation files telling Python how to install your project.\n",
      "Version control config, such as git. You should make a habit of\n",
      "using some type of version control for all your projects, no matter\n",
      "the size.\n",
      "Any other project files you might add in the future.\n",
      "\n",
      "By the end, your project layout will look like this:\n",
      "/home/user/Projects/flask-tutorial\n",
      "├── flaskr/\n",
      "│   ├── __init__.py\n",
      "│   ├── db.py\n",
      "│   ├── schema.sql\n",
      "│   ├── auth.py\n",
      "│   ├── blog.py\n",
      "│   ├── templates/\n",
      "│   │   ├── base.html\n",
      "│   │   ├── auth/\n",
      "│   │   │   ├── login.html\n",
      "│   │   │   └── register.html\n",
      "│   │   └── blog/\n",
      "│   │       ├── create.html\n",
      "│   │       ├── index.html\n",
      "│   │       └── update.html\n",
      "│   └── static/\n",
      "│       └── style.css\n",
      "├── tests/\n",
      "│   ├── conftest.py\n",
      "│   ├── data.sql\n",
      "│   ├── test_factory.py\n",
      "│   ├── test_db.py\n",
      "│   ├── test_auth.py\n",
      "│   └── test_blog.py\n",
      "├── .venv/\n",
      "├── pyproject.toml\n",
      "└── MANIFEST.in\n",
      "\n",
      "\n",
      "If you’re using version control, the following files that are generated\n",
      "while running your project should be ignored. There may be other files\n",
      "based on the editor you use. In general, ignore files that you didn’t\n",
      "write. For example, with git:\n",
      "\n",
      ".gitignore¶\n",
      ".venv/\n",
      "\n",
      "*.pyc\n",
      "__pycache__/\n",
      "\n",
      "instance/\n",
      "\n",
      ".pytest_cache/\n",
      ".coverage\n",
      "htmlcov/\n",
      "\n",
      "dist/\n",
      "build/\n",
      "*.egg-info/\n",
      "\n",
      "\n",
      "\n",
      "Continue to Application Setup.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Navigation\n",
      "\n",
      "Overview\n",
      "\n",
      "Tutorial\n",
      "\n",
      "Previous: Tutorial\n",
      "Next: Application Setup\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Quick search\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "    © Copyright 2010 Pallets.\n",
      "      Created using Sphinx 8.0.2.\n",
      "    \n",
      "\n",
      "\n",
      " {'source': './data/test.html', 'title': 'Project Layout — Flask Documentation (3.0.x)'}\n"
     ]
    }
   ],
   "source": [
    "for doc in data:\n",
    "    print(doc.page_content, doc.metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9a258fa0-5d6e-4258-97c5-bd5a6b8c8999",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1e32d9c0-512a-476f-b43a-f42006e32d9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "html_txt = ''\n",
    "with open('./data/test.html', 'r') as f:\n",
    "    for line in f.readlines():\n",
    "        html_txt += line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "bea940fe-356b-4774-a715-664405469d11",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n\\n<html lang=\"en\" data-content_root=\"../../\">\\n  <head>\\n    <meta charset=\"utf-8\" />\\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\" />\\n  <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\\n    <title>Project Layout &#8212; Flask Documentation (3.0.x)</title>\\n    <link rel=\"stylesheet\" type=\"text/css\" href=\"../../_static/pygments.css?v=4f649999\" />\\n    <link rel=\"stylesheet\" type=\"text/css\" href=\"../../_static/flask.css?v=b87c8d14\" />\\n    <link rel=\"stylesheet\" type=\"text/css\" href=\"../../_static/tabs.css?v=a5c4661c\" />\\n    <link rel=\"stylesheet\" type=\"text/css\" href=\"/_/static/css/badge_only.css\" />\\n    <script src=\"../../_static/documentation_options.js?v=9f2e3481\"></script>\\n    <script src=\"../../_static/doctools.js?v=9a2dae69\"></script>\\n    <script src=\"../../_static/sphinx_highlight.js?v=dc90522c\"></script>\\n    <script async=\"async\" src=\"/_/static/javascript/readthedocs-doc-embed.js\"></script>\\n    <link rel=\"icon\" href=\"../../_static/shortcut-icon.png\"/>\\n    <link rel=\"index\" title=\"Index\" href=\"../../genindex/\" />\\n    <link rel=\"search\" title=\"Search\" href=\"../../search/\" />\\n    <link rel=\"next\" title=\"Application Setup\" href=\"../factory/\" />\\n    <link rel=\"prev\" title=\"Tutorial\" href=\"../\" /> \\n  \\n<!-- RTD Extra Head -->\\n\\n<link rel=\"stylesheet\" href=\"/_/static/css/readthedocs-doc-embed.css\" type=\"text/css\" />\\n\\n<script type=\"application/json\" id=\"READTHEDOCS_DATA\">{\"ad_free\": false, \"api_host\": \"https://readthedocs.org\", \"builder\": \"sphinx\", \"canonical_url\": null, \"docroot\": \"/docs/\", \"features\": {\"docsearch_disabled\": true}, \"global_analytics_code\": \"UA-17997319-1\", \"language\": \"en\", \"page\": \"tutorial/layout\", \"programming_language\": \"py\", \"project\": \"flask\", \"proxied_api_host\": \"/_\", \"source_suffix\": \".rst\", \"subprojects\": {}, \"theme\": \"flask\", \"user_analytics_code\": \"\", \"version\": \"3.0.x\"}</script>\\n\\n<!--\\nUsing this variable directly instead of using `JSON.parse` is deprecated.\\nThe READTHEDOCS_DATA global variable will be removed in the future.\\n-->\\n<script type=\"text/javascript\">\\nREADTHEDOCS_DATA = JSON.parse(document.getElementById(\\'READTHEDOCS_DATA\\').innerHTML);\\n</script>\\n\\n<script type=\"text/javascript\" src=\"/_/static/javascript/readthedocs-analytics.js\" async=\"async\"></script>\\n\\n<!-- end RTD <extrahead> -->\\n</head><body>\\n    <div class=\"related\" role=\"navigation\" aria-label=\"Related\">\\n      <h3>Navigation</h3>\\n      <ul>\\n        <li class=\"right\" style=\"margin-right: 10px\">\\n          <a href=\"../../genindex/\" title=\"General Index\"\\n             accesskey=\"I\">index</a></li>\\n        <li class=\"right\" >\\n          <a href=\"../../py-modindex/\" title=\"Python Module Index\"\\n             >modules</a> |</li>\\n        <li class=\"right\" >\\n          <a href=\"../factory/\" title=\"Application Setup\"\\n             accesskey=\"N\">next</a> |</li>\\n        <li class=\"right\" >\\n          <a href=\"../\" title=\"Tutorial\"\\n             accesskey=\"P\">previous</a> |</li>\\n        <li class=\"nav-item nav-item-0\"><a href=\"../../\">Flask Documentation (3.0.x)</a> &#187;</li>\\n          <li class=\"nav-item nav-item-1\"><a href=\"../\" accesskey=\"U\">Tutorial</a> &#187;</li>\\n        <li class=\"nav-item nav-item-this\"><a href=\"\">Project Layout</a></li> \\n      </ul>\\n    </div>  \\n\\n    <div class=\"document\">\\n      <div class=\"documentwrapper\">\\n        <div class=\"bodywrapper\">\\n          <div class=\"body\" role=\"main\">\\n            \\n  <section id=\"project-layout\">\\n<h1>Project Layout<a class=\"headerlink\" href=\"#project-layout\" title=\"Link to this heading\">¶</a></h1>\\n<p>Create a project directory and enter it:</p>\\n<div class=\"highlight-none notranslate\"><div class=\"highlight\"><pre><span></span>$ mkdir flask-tutorial\\n$ cd flask-tutorial\\n</pre></div>\\n</div>\\n<p>Then follow the <a class=\"reference internal\" href=\"../../installation/\"><span class=\"doc\">installation instructions</span></a> to set\\nup a Python virtual environment and install Flask for your project.</p>\\n<p>The tutorial will assume you’re working from the <code class=\"docutils literal notranslate\"><span class=\"pre\">flask-tutorial</span></code>\\ndirectory from now on. The file names at the top of each code block are\\nrelative to this directory.</p>\\n<hr class=\"docutils\" />\\n<p>A Flask application can be as simple as a single file.</p>\\n<div class=\"literal-block-wrapper docutils container\" id=\"id1\">\\n<div class=\"code-block-caption\"><span class=\"caption-text\"><code class=\"docutils literal notranslate\"><span class=\"pre\">hello.py</span></code></span><a class=\"headerlink\" href=\"#id1\" title=\"Link to this code\">¶</a></div>\\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span></span><span class=\"kn\">from</span> <span class=\"nn\">flask</span> <span class=\"kn\">import</span> <span class=\"n\">Flask</span>\\n\\n<span class=\"n\">app</span> <span class=\"o\">=</span> <span class=\"n\">Flask</span><span class=\"p\">(</span><span class=\"vm\">__name__</span><span class=\"p\">)</span>\\n\\n\\n<span class=\"nd\">@app</span><span class=\"o\">.</span><span class=\"n\">route</span><span class=\"p\">(</span><span class=\"s1\">&#39;/&#39;</span><span class=\"p\">)</span>\\n<span class=\"k\">def</span> <span class=\"nf\">hello</span><span class=\"p\">():</span>\\n    <span class=\"k\">return</span> <span class=\"s1\">&#39;Hello, World!&#39;</span>\\n</pre></div>\\n</div>\\n</div>\\n<p>However, as a project gets bigger, it becomes overwhelming to keep all\\nthe code in one file. Python projects use <em>packages</em> to organize code\\ninto multiple modules that can be imported where needed, and the\\ntutorial will do this as well.</p>\\n<p>The project directory will contain:</p>\\n<ul class=\"simple\">\\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">flaskr/</span></code>, a Python package containing your application code and\\nfiles.</p></li>\\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">tests/</span></code>, a directory containing test modules.</p></li>\\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">.venv/</span></code>, a Python virtual environment where Flask and other\\ndependencies are installed.</p></li>\\n<li><p>Installation files telling Python how to install your project.</p></li>\\n<li><p>Version control config, such as <a class=\"reference external\" href=\"https://git-scm.com/\">git</a>. You should make a habit of\\nusing some type of version control for all your projects, no matter\\nthe size.</p></li>\\n<li><p>Any other project files you might add in the future.</p></li>\\n</ul>\\n<p>By the end, your project layout will look like this:</p>\\n<div class=\"highlight-none notranslate\"><div class=\"highlight\"><pre><span></span>/home/user/Projects/flask-tutorial\\n├── flaskr/\\n│   ├── __init__.py\\n│   ├── db.py\\n│   ├── schema.sql\\n│   ├── auth.py\\n│   ├── blog.py\\n│   ├── templates/\\n│   │   ├── base.html\\n│   │   ├── auth/\\n│   │   │   ├── login.html\\n│   │   │   └── register.html\\n│   │   └── blog/\\n│   │       ├── create.html\\n│   │       ├── index.html\\n│   │       └── update.html\\n│   └── static/\\n│       └── style.css\\n├── tests/\\n│   ├── conftest.py\\n│   ├── data.sql\\n│   ├── test_factory.py\\n│   ├── test_db.py\\n│   ├── test_auth.py\\n│   └── test_blog.py\\n├── .venv/\\n├── pyproject.toml\\n└── MANIFEST.in\\n</pre></div>\\n</div>\\n<p>If you’re using version control, the following files that are generated\\nwhile running your project should be ignored. There may be other files\\nbased on the editor you use. In general, ignore files that you didn’t\\nwrite. For example, with git:</p>\\n<div class=\"literal-block-wrapper docutils container\" id=\"id2\">\\n<div class=\"code-block-caption\"><span class=\"caption-text\"><code class=\"docutils literal notranslate\"><span class=\"pre\">.gitignore</span></code></span><a class=\"headerlink\" href=\"#id2\" title=\"Link to this code\">¶</a></div>\\n<div class=\"highlight-none notranslate\"><div class=\"highlight\"><pre><span></span>.venv/\\n\\n*.pyc\\n__pycache__/\\n\\ninstance/\\n\\n.pytest_cache/\\n.coverage\\nhtmlcov/\\n\\ndist/\\nbuild/\\n*.egg-info/\\n</pre></div>\\n</div>\\n</div>\\n<p>Continue to <a class=\"reference internal\" href=\"../factory/\"><span class=\"doc\">Application Setup</span></a>.</p>\\n</section>\\n\\n\\n            <div class=\"clearer\"></div>\\n          </div>\\n        </div>\\n      </div>\\n  <span id=\"sidebar-top\"></span>\\n      <div class=\"sphinxsidebar\" role=\"navigation\" aria-label=\"Main\">\\n        <div class=\"sphinxsidebarwrapper\">\\n  \\n    \\n            <p class=\"logo\"><a href=\"../../\">\\n              <img class=\"logo\" src=\"../../_static/flask-vertical.png\" alt=\"Logo of Flask\"/>\\n            </a></p>\\n  \\n<h3>Navigation</h3>\\n<ul>\\n  <li><a href=\"../../\">Overview</a>\\n    <ul>\\n      <li><a href=\"../\">Tutorial</a>\\n        <ul>\\n          <li>Previous: <a href=\"../\" title=\"previous chapter\">Tutorial</a>\\n          <li>Next: <a href=\"../factory/\" title=\"next chapter\">Application Setup</a></ul>\\n      </li>\\n    </ul>\\n  </li>\\n</ul>\\n<search id=\"searchbox\" style=\"display: none\" role=\"search\">\\n  <h3 id=\"searchlabel\">Quick search</h3>\\n    <div class=\"searchformwrapper\">\\n    <form class=\"search\" action=\"../../search/\" method=\"get\">\\n      <input type=\"text\" name=\"q\" aria-labelledby=\"searchlabel\" autocomplete=\"off\" autocorrect=\"off\" autocapitalize=\"off\" spellcheck=\"false\"/>\\n      <input type=\"submit\" value=\"Go\" />\\n    </form>\\n    </div>\\n</search>\\n<script>document.getElementById(\\'searchbox\\').style.display = \"block\"</script><div id=\"ethical-ad-placement\"></div>\\n        </div>\\n      </div>\\n      <div class=\"clearer\"></div>\\n    </div>\\n  \\n    <div class=\"footer\" role=\"contentinfo\">\\n    &#169; Copyright 2010 Pallets.\\n      Created using <a href=\"https://www.sphinx-doc.org/\">Sphinx</a> 8.0.2.\\n    </div>\\n  <script src=\"../../_static/version_warning_offset.js\"></script>\\n\\n  </body>\\n</html>'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html_txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "d343bdcf-d533-4a3e-926b-f40ac64aecaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(html_txt, 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "f8771066-33de-4420-bbe6-dbbc3260119d",
   "metadata": {},
   "outputs": [],
   "source": [
    "code_content = soup.find_all('div', class_='highlight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "cdfdeaa3-18fd-4d3f-b7fb-dd7d0867e342",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ mkdir flask-tutorial\n",
      "$ cd flask-tutorial\n",
      "\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "from flask import Flask\n",
      "\n",
      "app = Flask(__name__)\n",
      "\n",
      "\n",
      "@app.route('/')\n",
      "def hello():\n",
      "    return 'Hello, World!'\n",
      "\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "/home/user/Projects/flask-tutorial\n",
      "├── flaskr/\n",
      "│   ├── __init__.py\n",
      "│   ├── db.py\n",
      "│   ├── schema.sql\n",
      "│   ├── auth.py\n",
      "│   ├── blog.py\n",
      "│   ├── templates/\n",
      "│   │   ├── base.html\n",
      "│   │   ├── auth/\n",
      "│   │   │   ├── login.html\n",
      "│   │   │   └── register.html\n",
      "│   │   └── blog/\n",
      "│   │       ├── create.html\n",
      "│   │       ├── index.html\n",
      "│   │       └── update.html\n",
      "│   └── static/\n",
      "│       └── style.css\n",
      "├── tests/\n",
      "│   ├── conftest.py\n",
      "│   ├── data.sql\n",
      "│   ├── test_factory.py\n",
      "│   ├── test_db.py\n",
      "│   ├── test_auth.py\n",
      "│   └── test_blog.py\n",
      "├── .venv/\n",
      "├── pyproject.toml\n",
      "└── MANIFEST.in\n",
      "\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      ".venv/\n",
      "\n",
      "*.pyc\n",
      "__pycache__/\n",
      "\n",
      "instance/\n",
      "\n",
      ".pytest_cache/\n",
      ".coverage\n",
      "htmlcov/\n",
      "\n",
      "dist/\n",
      "build/\n",
      "*.egg-info/\n",
      "\n",
      "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n"
     ]
    }
   ],
   "source": [
    "for ele in code_content:\n",
    "    print(ele.text)\n",
    "    print(\"+\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6bf43da-6958-4d3a-99f7-fbf017dcace2",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "3edc9a08-b828-4741-83bf-fda121d2ca0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyMuPDFLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "864e0656-a6c5-4e4b-9dc7-c568b341f320",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = PyMuPDFLoader('./data/zhidu_travel.pdf')\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "e7555376-918e-42ac-b241-9837d91e1d1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='1\n",
      "差旅费用标准\n",
      "差旅费开支范围包括工作人员临时到常驻地以外地区\n",
      "公务出差所发生的城市间交通费、住宿费、伙食补助费和市\n",
      "内交通费。\n",
      "一、城市间交通费\n",
      "城市间交通费是指工作人员因公到常驻地以外地区出\n",
      "差乘坐火车、轮船、飞机等交通工具所发生的费用。\n",
      "1.出差人员在不影响公务、确保安全的前提下，选乘经\n",
      "济便捷的交通工具。\n",
      "2.出差人员要按照规定等级乘坐交通工具，未按规定乘\n",
      "坐的，超支部分自理。乘坐交通工具的等级见下表：\n",
      "级别\n",
      "火车\n",
      "（含高铁、动车、全列软席列车）\n",
      "轮船\n",
      "（不包括\n",
      "旅游船）\n",
      "飞机\n",
      "其他交通工具\n",
      "（不包括出租\n",
      "小汽车）\n",
      "享受副部级\n",
      "待遇及以上\n",
      "人员\n",
      "火车软席（软座、软卧）\n",
      "，高铁/动车商\n",
      "务座，全列软席列车一等软座\n",
      "一等舱\n",
      "头等舱\n",
      "凭据报销\n",
      "秘书长及副\n",
      "秘书长\n",
      "火车软席（软座、软卧）\n",
      "，高铁/动车一\n",
      "等座，全列软席列车一等软座\n",
      "二等舱\n",
      "经济舱\n",
      "凭据报销\n",
      "其余人员\n",
      "火车硬席（硬座、硬卧）\n",
      "，高铁/动车二\n",
      "等座、全列软席列车二等软座\n",
      "三等舱\n",
      "经济舱\n",
      "凭据报销\n",
      "' metadata={'source': './data/zhidu_travel.pdf', 'file_path': './data/zhidu_travel.pdf', 'page': 0, 'total_pages': 3, 'format': 'PDF 1.7', 'title': '', 'author': 'gy', 'subject': '', 'keywords': '', 'creator': 'WPS 文字', 'producer': '', 'creationDate': \"D:20200401170344+09'03'\", 'modDate': \"D:20200401170344+09'03'\", 'trapped': ''}\n",
      "====================================================================================================\n",
      "page_content='2\n",
      "备注：\n",
      "享受副部级待遇及以上人员出差，因工作需要，随行一\n",
      "人可乘坐同等级交通工具；乘坐飞机的，民航发展基金、燃\n",
      "油附加费可以凭据报销；乘坐飞机、火车、轮船等交通工具\n",
      "的，每人次可以购买交通意外保险一份。由我会统一购买交\n",
      "通意外保险的，不再重复购买。\n",
      "3.我会工作人员出差，高铁最短5 个小时内能够到达目\n",
      "的地的，原则上应乘坐高铁，如遇特殊情况，经分管副秘书\n",
      "长、秘书长审批后，可乘坐飞机。\n",
      "4.国内出差购买机票,应当按照厉行节约和支持本国航\n",
      "空公司发展的原则,优先购买通过政府采购方式确定的我国\n",
      "航空公司航班优惠机票。飞机票应通过定点供应商采购。\n",
      "二、住宿费\n",
      "住宿费是指工作人员因公出差期间入住宾馆（包括饭店、\n",
      "招待所，下同）发生的房租费用。享受副部级待遇及以上人\n",
      "员住普通套间，秘书长及以下人员住单间或标准间。出差人\n",
      "员应当在职务级别对应的住宿费标准限额内，选择安全、经\n",
      "济、便捷的宾馆住宿。国内出差发生的住宿，原则上应当按\n",
      "照住宿费标准执行，如遇特殊情况（如对方组织单位指定或\n",
      "推荐酒店等情况）\n",
      "，超出费用标准的，须事先报分管资金的\n",
      "副秘书长、秘书长批准后方可报销。\n",
      "住宿费按照财政部《中央和国家机关工作人员赴地方差\n",
      "旅住宿费标准明细表》\n",
      "（财行[2016]71 号）执行。\n",
      "' metadata={'source': './data/zhidu_travel.pdf', 'file_path': './data/zhidu_travel.pdf', 'page': 1, 'total_pages': 3, 'format': 'PDF 1.7', 'title': '', 'author': 'gy', 'subject': '', 'keywords': '', 'creator': 'WPS 文字', 'producer': '', 'creationDate': \"D:20200401170344+09'03'\", 'modDate': \"D:20200401170344+09'03'\", 'trapped': ''}\n",
      "====================================================================================================\n",
      "page_content='3\n",
      "三、伙食补助费\n",
      "伙食补助费是指对工作人员在因公出差期间给予的伙\n",
      "食补助费用。伙食补助费按出差自然(日历)天数计算，按规\n",
      "定标准包干使用。出差人员应当自行用餐。凡由接待单位统\n",
      "一安排用餐的，不再发放伙食补助费。\n",
      "伙食补助费参考以下标准：\n",
      "地区\n",
      "伙食补助费标准\n",
      "西藏、青海、新疆\n",
      "120 元/人、天\n",
      "其他省份\n",
      "100 元/人、天\n",
      "四、市内交通费\n",
      "市内交通费是指工作人员因公出差期间发生的市内交\n",
      "通费用。出差人员的市内交通费按出差的自然（日历）天数\n",
      "计算，每人每天80 元包干使用。由接待单位或其他单位提\n",
      "供交通工具的，不再发放市内交通费。\n",
      "' metadata={'source': './data/zhidu_travel.pdf', 'file_path': './data/zhidu_travel.pdf', 'page': 2, 'total_pages': 3, 'format': 'PDF 1.7', 'title': '', 'author': 'gy', 'subject': '', 'keywords': '', 'creator': 'WPS 文字', 'producer': '', 'creationDate': \"D:20200401170344+09'03'\", 'modDate': \"D:20200401170344+09'03'\", 'trapped': ''}\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for doc in docs:\n",
    "    print(doc)\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e920f85-7c8a-4bde-a1c9-9b4789fca77f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "07e5ed5d-5cba-4238-930f-725aae809ae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import fitz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "15e6796e-6d7e-4f2b-ab56-c33545da4f5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = fitz.open('./data/zhidu_travel.pdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ceb83f70-b350-4ce2-917c-3df2f5e3cdd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_data = []\n",
    "text_data = []\n",
    "\n",
    "\n",
    "doc_tables = {}\n",
    "for idx, page in enumerate(doc):\n",
    "    text = page.get_text()\n",
    "    text_data.append(text)\n",
    "    tabs = page.find_tables()\n",
    "    for i,tab in enumerate(tabs):\n",
    "        ds = tab.to_pandas()\n",
    "        table_data.append(ds.to_markdown())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "3bd1b9c2-6b56-4bbf-bd48-de911ea9a003",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|    | 级别       | 火车                                | 轮船       | 飞机   | 其他交通工具   |\n",
      "|    |            | （含高铁、动车、全列软席列车）      | （不包括   |        | （不包括出租   |\n",
      "|    |            |                                     | 旅游船）   |        | 小汽车）       |\n",
      "|---:|:-----------|:------------------------------------|:-----------|:-------|:---------------|\n",
      "|  0 | 享受副部级 | 火车软席（软座、软卧），高铁/动车商 | 一等舱     | 头等舱 | 凭据报销       |\n",
      "|    | 待遇及以上 | 务座，全列软席列车一等软座          |            |        |                |\n",
      "|    | 人员       |                                     |            |        |                |\n",
      "|  1 | 秘书长及副 | 火车软席（软座、软卧），高铁/动车一 | 二等舱     | 经济舱 | 凭据报销       |\n",
      "|    | 秘书长     | 等座，全列软席列车一等软座          |            |        |                |\n",
      "|  2 | 其余人员   | 火车硬席（硬座、硬卧），高铁/动车二 | 三等舱     | 经济舱 | 凭据报销       |\n",
      "|    |            | 等座、全列软席列车二等软座          |            |        |                |\n",
      "====================================================================================================\n",
      "|    | 地区             | 伙食补助费标准   |\n",
      "|---:|:-----------------|:-----------------|\n",
      "|  0 | 西藏、青海、新疆 | 120 元/人、天    |\n",
      "|  1 | 其他省份         | 100 元/人、天    |\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for tab in table_data:\n",
    "    print(tab)\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "862d21e2-019d-4b11-9ad6-46b05706bf78",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "97d90ffd-c720-44ba-a233-4573d45c669c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4.Unstructured\n",
    "unstructured是一个Python库，主要用于处理非结构化数据，如文本、PDF文档、电子邮件等。这个库可以帮助你从这些非结构化的数据源中提取信息\n",
    "\n",
    "- pip install -i https://pypi.tuna.tsinghua.edu.cn/simple unstructured\n",
    "- pip install -i https://pypi.tuna.tsinghua.edu.cn/simple langchain_unstructured\n",
    "- unstructured文档： https://docs.unstructured.io/open-source/core-functionality/partitioning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "83269dee-42cc-4a84-820b-49a06c67c9f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_unstructured import UnstructuredLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "efd6cfec-aabe-4115-a8f3-9985643db838",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = UnstructuredLoader('./data/zhidu_travel.pdf')\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1961c43e-376c-4aa1-a8e7-e1bb6e834b99",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='差旅费用标准' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((261.48, 78.601), (261.48, 96.601), (369.83865000000003, 96.601), (369.83865000000003, 78.601)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '4ec459826ffa4bc0dacd148f86613a84'}\n",
      "====================================================================================================\n",
      "page_content='差旅费开支范围包括工作人员临时到常驻地以外地区' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 143.55579999999998), (122.04, 159.50580000000002), (505.30910000000006, 159.50580000000002), (505.30910000000006, 143.55579999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'd50e809619c66ec3820c168f6515eae1'}\n",
      "====================================================================================================\n",
      "page_content='公务出差所发生的城市间交通费、住宿费、伙食补助费和市' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 171.5158), (90.0, 187.46580000000006), (505.18700000000007, 187.46580000000006), (505.18700000000007, 171.5158)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '05afc8e2928ca098aac7756d0c4a22fe'}\n",
      "====================================================================================================\n",
      "page_content='内交通费。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 199.47479999999996), (90.0, 215.4248), (169.90789999999998, 215.4248), (169.90789999999998, 199.47479999999996)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'ecf4cdcecfc5ddd5b78496129b044e14'}\n",
      "====================================================================================================\n",
      "page_content='一、 城市间交通费' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 227.42819999999995), (122.04, 243.3782), (254.02759999999995, 243.3782), (254.02759999999995, 227.42819999999995)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '7bad53ba226eaeabd520f3d706479196'}\n",
      "====================================================================================================\n",
      "page_content='城市间交通费是指工作人员因公到常驻地以外地区出' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 255.5158), (122.04, 271.46580000000006), (505.3043000000003, 271.46580000000006), (505.3043000000003, 255.5158)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '00c9698ce1fb11970d2bab7d3b7b1301'}\n",
      "====================================================================================================\n",
      "page_content='差乘坐火车、轮船、飞机等交通工具所发生的费用。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 283.4748000000001), (90.0, 299.4248000000001), (457.8989000000001, 299.4248000000001), (457.8989000000001, 283.4748000000001)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'de0eb21f34efb585b244fd6853df324f'}\n",
      "====================================================================================================\n",
      "page_content='1.出差人员在不影响公务、确保安全的前提下，选乘经' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 311.5558), (122.04, 327.5058), (505.3091, 327.5058), (505.3091, 311.5558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '076619365975f463bfae2009fb7aa790'}\n",
      "====================================================================================================\n",
      "page_content='济便捷的交通工具。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 339.5158), (90.0, 355.4658), (233.98609999999996, 355.4658), (233.98609999999996, 339.5158)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'd49a5f6b3633201f1d43a53af0e2343e'}\n",
      "====================================================================================================\n",
      "page_content='2.出差人员要按照规定等级乘坐交通工具，未按规定乘' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 367.4748), (122.04, 383.4248), (505.1879, 383.4248), (505.1879, 367.4748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'b856306ac8fc8f3cdae91965444852fa'}\n",
      "====================================================================================================\n",
      "page_content='坐的，超支部分自理。乘坐交通工具的等级见下表：' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 395.5558), (90.0, 411.50579999999997), (457.8989000000001, 411.50579999999997), (457.8989000000001, 395.5558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '30c3f17b5757dace2e3da6ebbebbea5e'}\n",
      "====================================================================================================\n",
      "page_content='轮船' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((348.12, 421.4248), (348.12, 431.8748), (369.1294, 431.8748), (369.1294, 421.4248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '90f7b2ac6cdd1d4b4ca982ab9d4138e4'}\n",
      "====================================================================================================\n",
      "page_content='其他交通工具' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((450.0, 421.4248), (450.0, 431.8748), (513.12745, 431.8748), (513.12745, 421.4248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'fe04404c2d51eea8040e9b590f4d1c5d'}\n",
      "====================================================================================================\n",
      "page_content='火车' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((222.48, 433.0638), (222.48, 443.5138), (243.4894, 443.5138), (243.4894, 433.0638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'c4229602ee6a4ac95d8ea4cda017ace9'}\n",
      "====================================================================================================\n",
      "page_content='级别' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((95.04, 444.8248), (95.04, 455.27479999999997), (116.0494, 455.27479999999997), (116.0494, 444.8248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '2ab4fb9a56e6a13cfc91da7783d1d010'}\n",
      "====================================================================================================\n",
      "page_content='（不包括' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((337.68, 444.8248), (337.68, 455.27479999999997), (379.80819999999994, 455.27479999999997), (379.80819999999994, 444.8248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'a8e2032568c114f6986f82ba9dfe5975'}\n",
      "====================================================================================================\n",
      "page_content='飞机' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((404.64, 444.8248), (404.64, 455.27479999999997), (425.64939999999996, 455.27479999999997), (425.64939999999996, 444.8248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '3cdefeb60b7cfddca005772963d29cab'}\n",
      "====================================================================================================\n",
      "page_content='（不包括出租' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((450.0, 444.8248), (450.0, 455.27479999999997), (513.12745, 455.27479999999997), (513.12745, 444.8248)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'e5d7330ceb3919ffb671a504959e30a3'}\n",
      "====================================================================================================\n",
      "page_content='（含高铁、动车、全列软席列车）' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((154.2, 456.4638), (154.2, 466.9138), (312.2425, 466.9138), (312.2425, 456.4638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '24087b7881b238ce26b8a3b566a406ee'}\n",
      "====================================================================================================\n",
      "page_content='旅游船）' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((337.68, 468.2238), (337.68, 478.67379999999997), (379.80819999999994, 478.67379999999997), (379.80819999999994, 468.2238)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '9ea6dfee39657f84ddd6ed187d8de6f2'}\n",
      "====================================================================================================\n",
      "page_content='小汽车）' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((460.559, 468.2238), (460.559, 478.67379999999997), (502.68719999999996, 478.67379999999997), (502.68719999999996, 468.2238)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '8774d059d451174bb48fce8fbf64bd07'}\n",
      "====================================================================================================\n",
      "page_content='享受副部级' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((79.32, 501.5848), (79.32, 512.0347999999999), (131.76925, 512.0347999999999), (131.76925, 501.5848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '3f12895d1406553c4026543fbc6efb28'}\n",
      "====================================================================================================\n",
      "page_content='火车软席（软座、软卧），高铁/动车商' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((143.64, 513.2238), (143.64, 523.6738), (322.08924999999994, 523.6738), (322.08924999999994, 513.2238)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'c7a632c0677d21f155c022735d9a9213'}\n",
      "====================================================================================================\n",
      "page_content='待遇及以上' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((79.32, 524.9848), (79.32, 535.4348), (131.7685, 535.4348), (131.7685, 524.9848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '364f7f5cf423d02875b59f344b88105f'}\n",
      "====================================================================================================\n",
      "page_content='一等舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((342.84, 524.9848), (342.84, 535.4348), (374.2892499999999, 535.4348), (374.2892499999999, 524.9848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'efd245763dd50c73e80515e65ceb0f98'}\n",
      "====================================================================================================\n",
      "page_content='头等舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((399.359, 524.9848), (399.359, 535.4348), (430.80824999999993, 535.4348), (430.80824999999993, 524.9848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'bdaa65665337b2ee58319a09d2be1833'}\n",
      "====================================================================================================\n",
      "page_content='凭据报销' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((460.559, 524.9848), (460.559, 535.4348), (502.44809999999995, 535.4348), (502.44809999999995, 524.9848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '91363071866be6f41ec2e492ede2cf19'}\n",
      "====================================================================================================\n",
      "page_content='务座，全列软席列车一等软座' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((164.64, 536.6238), (164.64, 547.0737999999999), (301.08549999999997, 547.0737999999999), (301.08549999999997, 536.6238)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '7890b21df5ef3abc128854b426161be2'}\n",
      "====================================================================================================\n",
      "page_content='人员' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((95.04, 548.3848), (95.04, 558.8348000000001), (115.92985, 558.8348000000001), (115.92985, 548.3848)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '585d7fa826e13d5604d4e388bed6f717'}\n",
      "====================================================================================================\n",
      "page_content='秘书长及副' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((79.32, 602.8638), (79.32, 613.3137999999999), (131.7685, 613.3137999999999), (131.7685, 602.8638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'dc11ec87314159e0f4047c26a82b5e50'}\n",
      "====================================================================================================\n",
      "page_content='火车软席（软座、软卧），高铁/动车一' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((143.64, 602.8638), (143.64, 613.3137999999999), (322.08924999999994, 613.3137999999999), (322.08924999999994, 602.8638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'd74cecfb4373fadea34d194bb83af8c4'}\n",
      "====================================================================================================\n",
      "page_content='二等舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((342.84, 614.5048), (342.84, 624.9548), (374.2892499999999, 624.9548), (374.2892499999999, 614.5048)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '352e7b823e0d9b833bd8220d3a26ce4c'}\n",
      "====================================================================================================\n",
      "page_content='经济舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((399.359, 614.5048), (399.359, 624.9548), (430.80824999999993, 624.9548), (430.80824999999993, 614.5048)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '9ee724ae2abdee363dd561a1557d674a'}\n",
      "====================================================================================================\n",
      "page_content='凭据报销' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((460.559, 614.5048), (460.559, 624.9548), (502.44809999999995, 624.9548), (502.44809999999995, 614.5048)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'd2c96b273ea6507fd27377847a03077f'}\n",
      "====================================================================================================\n",
      "page_content='秘书长' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((89.759, 626.2638), (89.759, 636.7138), (121.20824999999999, 636.7138), (121.20824999999999, 626.2638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '7b6eb330b974d867974244a18a005912'}\n",
      "====================================================================================================\n",
      "page_content='等座，全列软席列车一等软座' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((164.64, 626.2638), (164.64, 636.7138), (301.08549999999997, 636.7138), (301.08549999999997, 626.2638)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '8b96ca39490fa0ec9e93a7c7a3a59b61'}\n",
      "====================================================================================================\n",
      "page_content='火车硬席（硬座、硬卧），高铁/动车二' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((143.64, 692.5048), (143.64, 702.9548), (322.08924999999994, 702.9548), (322.08924999999994, 692.5048)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '09e72764feb1a8aa25ef07406823dade'}\n",
      "====================================================================================================\n",
      "page_content='其余人员' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((84.6, 704.1448), (84.6, 714.5948000000001), (126.4891, 714.5948000000001), (126.4891, 704.1448)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '39aa752718750c9e2383b6146eed1848'}\n",
      "====================================================================================================\n",
      "page_content='三等舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((342.84, 704.1448), (342.84, 714.5948000000001), (374.2892499999999, 714.5948000000001), (374.2892499999999, 704.1448)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'ae8357482d939edf2de6c31b493d7fd8'}\n",
      "====================================================================================================\n",
      "page_content='经济舱' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((399.359, 704.1448), (399.359, 714.5948000000001), (430.80824999999993, 714.5948000000001), (430.80824999999993, 704.1448)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '0d9ef9fdc9db3f1767919e0064d25e3a'}\n",
      "====================================================================================================\n",
      "page_content='凭据报销' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((460.559, 704.1448), (460.559, 714.5948000000001), (502.44809999999995, 714.5948000000001), (502.44809999999995, 704.1448)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '009d819e8a006af363c194a01741cebc'}\n",
      "====================================================================================================\n",
      "page_content='等座、全列软席列车二等软座' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((164.64, 715.9048), (164.64, 726.3548000000001), (301.08549999999997, 726.3548000000001), (301.08549999999997, 715.9048)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '56c0f430276bc23656c821ccdb013426'}\n",
      "====================================================================================================\n",
      "page_content='1' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((295.32, 772.613), (295.32, 781.613), (299.87399999999997, 781.613), (299.87399999999997, 772.613)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'parent_id': '56c0f430276bc23656c821ccdb013426', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': '4aa2a60226025b5ea90c2866a3e6b1e7'}\n",
      "====================================================================================================\n",
      "page_content='备注：' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 81.1558), (90.0, 97.10580000000004), (137.9891, 97.10580000000004), (137.9891, 81.1558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'b5e4077064a3e2166e3e17d3248acbc6'}\n",
      "====================================================================================================\n",
      "page_content='享受副部级待遇及以上人员出差，因工作需要，随行一' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 109.11580000000004), (122.04, 125.06580000000008), (505.3091, 125.06580000000008), (505.3091, 109.11580000000004)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '118ea3e47f2a045990b363b89a9d3322'}\n",
      "====================================================================================================\n",
      "page_content='人可乘坐同等级交通工具；乘坐飞机的，民航发展基金、燃' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 137.07479999999998), (90.0, 153.02480000000003), (505.30899999999997, 153.02480000000003), (505.30899999999997, 137.07479999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '6a22bc997e75afcbe31df42c118c66e3'}\n",
      "====================================================================================================\n",
      "page_content='油附加费可以凭据报销；乘坐飞机、火车、轮船等交通工具' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 165.1558), (90.0, 181.10580000000004), (505.18700000000007, 181.10580000000004), (505.18700000000007, 165.1558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '9aaa803beb133ce9cca9c3db8d65e108'}\n",
      "====================================================================================================\n",
      "page_content='的，每人次可以购买交通意外保险一份。由我会统一购买交' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 193.11579999999992), (90.0, 209.06579999999997), (505.18640000000005, 209.06579999999997), (505.18640000000005, 193.11579999999992)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '29459cc16f343eea84b1d75d79c79c2a'}\n",
      "====================================================================================================\n",
      "page_content='通意外保险的，不再重复购买。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 221.07479999999998), (90.0, 237.02480000000003), (313.9034, 237.02480000000003), (313.9034, 221.07479999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '2f58ba07438ed4e72433079b69e0d9d3'}\n",
      "====================================================================================================\n",
      "page_content='3.我会工作人员出差，高铁最短 5 个小时内能够到达目' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 248.07479999999998), (122.04, 264.0248), (505.3061000000001, 264.0248), (505.3061000000001, 248.07479999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '281843b61eabc9374b12a141997073f9'}\n",
      "====================================================================================================\n",
      "page_content='的地的，原则上应乘坐高铁，如遇特殊情况，经分管副秘书' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 279.2748), (90.0, 295.2248000000001), (505.3091, 295.2248000000001), (505.3091, 279.2748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'ca11f356b9838f365aec23cbc172d45c'}\n",
      "====================================================================================================\n",
      "page_content='长、秘书长审批后，可乘坐飞机。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 310.4748000000001), (90.0, 326.4248000000001), (329.984, 326.4248000000001), (329.984, 310.4748000000001)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '6cf8dbb684309fec13cf0bc8b95e91d1'}\n",
      "====================================================================================================\n",
      "page_content='4.国内出差购买机票,应当按照厉行节约和支持本国航' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 341.6748), (122.04, 357.6248), (505.1849499999999, 357.6248), (505.1849499999999, 341.6748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'd0eb3e5649dea3ff3b1045e5f71ff2f9'}\n",
      "====================================================================================================\n",
      "page_content='空公司发展的原则,优先购买通过政府采购方式确定的我国' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 372.8748), (90.0, 388.8248), (505.1862999999999, 388.8248), (505.1862999999999, 372.8748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '5f457041cc34df59b3e224b0ae1c6f0d'}\n",
      "====================================================================================================\n",
      "page_content='航空公司航班优惠机票。飞机票应通过定点供应商采购。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 404.0748), (90.0, 420.02479999999997), (489.9380000000001, 420.02479999999997), (489.9380000000001, 404.0748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'a17ff12722e33011a0faf72c7ddb7429'}\n",
      "====================================================================================================\n",
      "page_content='二、住宿费' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 435.1472), (122.04, 451.0972), (201.94789999999998, 451.0972), (201.94789999999998, 435.1472)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '2ee998cd70ab59e1263ab8c4616a7374'}\n",
      "====================================================================================================\n",
      "page_content='住宿费是指工作人员因公出差期间入住宾馆（包括饭店、' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 466.4748), (122.04, 482.4248), (521.1479, 482.4248), (521.1479, 466.4748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '9894efbd8a6d2bfedf43753302888168'}\n",
      "====================================================================================================\n",
      "page_content='招待所，下同）发生的房租费用。享受副部级待遇及以上人' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 497.6748), (90.0, 513.6248), (505.3061000000001, 513.6248), (505.3061000000001, 497.6748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '59ab6c49cbe2b7816944b67f0318f211'}\n",
      "====================================================================================================\n",
      "page_content='员住普通套间，秘书长及以下人员住单间或标准间。出差人' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 528.8747999999999), (90.0, 544.8247999999999), (505.3091, 544.8247999999999), (505.3091, 528.8747999999999)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '06bcdaaea0b6c38a4a95cc642150b9e5'}\n",
      "====================================================================================================\n",
      "page_content='员应当在职务级别对应的住宿费标准限额内，选择安全、经' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 560.0748), (90.0, 576.0247999999999), (505.19, 576.0247999999999), (505.19, 560.0748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'e5be3185c8c667fdbb635f15470c28b6'}\n",
      "====================================================================================================\n",
      "page_content='济、便捷的宾馆住宿。国内出差发生的住宿，原则上应当按' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 591.2748), (90.0, 607.2248000000001), (505.30760000000004, 607.2248000000001), (505.30760000000004, 591.2748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '51c9c606e6e93e39e6d0f5ccc584a2e9'}\n",
      "====================================================================================================\n",
      "page_content='照住宿费标准执行，如遇特殊情况（如对方组织单位指定或' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 622.4748), (90.0, 638.4248), (505.1855000000001, 638.4248), (505.1855000000001, 622.4748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'da71299cd9386ebe2190ae1f2739d442'}\n",
      "====================================================================================================\n",
      "page_content='推荐酒店等情况），超出费用标准的，须事先报分管资金的' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 653.6748), (90.0, 669.6248), (505.1841499999999, 669.6248), (505.1841499999999, 653.6748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '344d98d4c79ee300ba33f0aa13f228d3'}\n",
      "====================================================================================================\n",
      "page_content='副秘书长、秘书长批准后方可报销。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 684.8747999999999), (90.0, 700.8248), (345.9449, 700.8248), (345.9449, 684.8747999999999)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'da15fa18045a3e38b75bc4091b5d07c6'}\n",
      "====================================================================================================\n",
      "page_content='住宿费按照财政部《中央和国家机关工作人员赴地方差' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 718.1158), (122.04, 734.0658000000001), (505.30310000000014, 734.0658000000001), (505.30310000000014, 718.1158)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '3ac37fd7b10004a0dd5853fab08b46c1'}\n",
      "====================================================================================================\n",
      "page_content='旅住宿费标准明细表》（财行[2016]71 号）执行。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 746.0748), (90.0, 762.0248), (437.86690000000004, 762.0248), (437.86690000000004, 746.0748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'a4deef16ab62779012af972d8b566b1b'}\n",
      "====================================================================================================\n",
      "page_content='2' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((295.32, 772.613), (295.32, 781.613), (299.87399999999997, 781.613), (299.87399999999997, 772.613)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 2, 'parent_id': 'a4deef16ab62779012af972d8b566b1b', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': '28a5c2d7d7c922422d20314fdc7aa72f'}\n",
      "====================================================================================================\n",
      "page_content='三、伙食补助费' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 81.02819999999997), (122.04, 96.97820000000002), (233.98699999999997, 96.97820000000002), (233.98699999999997, 81.02819999999997)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '5e40f836f927514becc25321643b55ad'}\n",
      "====================================================================================================\n",
      "page_content='伙食补助费是指对工作人员在因公出差期间给予的伙' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 109.11580000000004), (122.04, 125.06580000000008), (505.3043000000003, 125.06580000000008), (505.3043000000003, 109.11580000000004)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '585d394ffaf4444580b249474d84ef46'}\n",
      "====================================================================================================\n",
      "page_content='食补助费用。伙食补助费按出差自然(日历)天数计算，按规' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 137.07479999999998), (90.0, 153.02480000000003), (505.1894, 153.02480000000003), (505.1894, 137.07479999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '217bdc04dc74c9b8b728b2568ad65dfd'}\n",
      "====================================================================================================\n",
      "page_content='定标准包干使用。出差人员应当自行用餐。凡由接待单位统' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 165.1558), (90.0, 181.10580000000004), (505.18700000000007, 181.10580000000004), (505.18700000000007, 165.1558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '50f4e40f15130bb27599d30a7e0e15ef'}\n",
      "====================================================================================================\n",
      "page_content='一安排用餐的，不再发放伙食补助费。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 193.11579999999992), (90.0, 209.06579999999997), (361.9019, 209.06579999999997), (361.9019, 193.11579999999992)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '393b1ab179b1ae7c0d5a0d93b2abac2a'}\n",
      "====================================================================================================\n",
      "page_content='伙食补助费参考以下标准：' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 221.07479999999998), (122.04, 237.02480000000003), (314.02459999999996, 237.02480000000003), (314.02459999999996, 221.07479999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'a75d736ef8be9a1affba16270ddf16da'}\n",
      "====================================================================================================\n",
      "page_content='地区' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((164.4, 252.69799999999998), (164.4, 264.698), (188.4, 264.698), (188.4, 252.69799999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'eac89438e381751a42ccbe8a4efff769'}\n",
      "====================================================================================================\n",
      "page_content='伙食补助费标准' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((378.12, 252.69799999999998), (378.12, 264.698), (462.12, 264.698), (462.12, 252.69799999999998)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'ff1b3af1b25979e5383b9da5df459bcd'}\n",
      "====================================================================================================\n",
      "page_content='西藏、青海、新疆' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((128.4, 281.13699999999994), (128.4, 293.13699999999994), (224.4, 293.13699999999994), (224.4, 281.13699999999994)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '37b69807d9208b80ce7818f0a001afcf'}\n",
      "====================================================================================================\n",
      "page_content='120 元/人、天' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((382.68, 281.13699999999994), (382.68, 293.13699999999994), (457.68, 293.13699999999994), (457.68, 281.13699999999994)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'parent_id': '37b69807d9208b80ce7818f0a001afcf', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': '62c796a400f610236db8f5950d9bfc74'}\n",
      "====================================================================================================\n",
      "page_content='其他省份' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((152.4, 309.698), (152.4, 321.698), (200.4, 321.698), (200.4, 309.698)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '197fdda210c4d3890caf013d8567f497'}\n",
      "====================================================================================================\n",
      "page_content='100 元/人、天' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((382.68, 309.698), (382.68, 321.698), (457.68, 321.698), (457.68, 309.698)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'parent_id': '197fdda210c4d3890caf013d8567f497', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd603c550b79836145bfcc4aa3e9692bd'}\n",
      "====================================================================================================\n",
      "page_content='四、市内交通费' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 334.9472), (122.04, 350.8972), (233.86689999999996, 350.8972), (233.86689999999996, 334.9472)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'c7a18c80e4711a51422bf8c15c80b90b'}\n",
      "====================================================================================================\n",
      "page_content='市内交通费是指工作人员因公出差期间发生的市内交' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((122.04, 363.1558), (122.04, 379.1058), (505.3043000000003, 379.1058), (505.3043000000003, 363.1558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'cc3863780a5ab73cf0050d22d371a634'}\n",
      "====================================================================================================\n",
      "page_content='通费用。出差人员的市内交通费按出差的自然（日历）天数' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 391.1158), (90.0, 407.06579999999997), (505.1894, 407.06579999999997), (505.1894, 391.1158)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'e0db5fcbdee12cf9711cec13baab0246'}\n",
      "====================================================================================================\n",
      "page_content='计算，每人每天 80 元包干使用。由接待单位或其他单位提' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 419.0748), (90.0, 435.02479999999997), (505.1826999999999, 435.02479999999997), (505.1826999999999, 419.0748)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'b7fc637837d23c87af654f67653fc350'}\n",
      "====================================================================================================\n",
      "page_content='供交通工具的，不再发放市内交通费。' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((90.0, 447.1558), (90.0, 463.1058), (361.9019, 463.1058), (361.9019, 447.1558)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': 'df3dc5dc314848d1565ff844f4ec5fd8'}\n",
      "====================================================================================================\n",
      "page_content='3' metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((295.32, 772.613), (295.32, 781.613), (299.87399999999997, 781.613), (299.87399999999997, 772.613)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 3, 'parent_id': 'df3dc5dc314848d1565ff844f4ec5fd8', 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'f8500ed10fe73da94d740df1c25ca815'}\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for doc in docs:\n",
    "    print(doc)\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "f175a704-64f3-46d0-88b8-d479b31fde04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': './data/zhidu_travel.pdf', 'coordinates': {'points': ((261.48, 78.601), (261.48, 96.601), (369.83865000000003, 96.601), (369.83865000000003, 78.601)), 'system': 'PixelSpace', 'layout_width': 595.3, 'layout_height': 841.9}, 'file_directory': './data', 'filename': 'zhidu_travel.pdf', 'languages': ['eng'], 'last_modified': '2023-06-05T11:30:43', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'Title', 'element_id': '4ec459826ffa4bc0dacd148f86613a84'}, page_content='差旅费用标准')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61348d7e-5890-404a-9cd5-cd7ef82756f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f1f32042-7cc4-4d3c-b19d-2f05a07df7d7",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 5.ppt word excel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2cb5397-188c-41f7-a812-6ba82b13e116",
   "metadata": {},
   "source": [
    "#### ppt\n",
    "- pip install python-pptx \n",
    "- yum install libreoffice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "af41a8d5-da44-4987-abec-e068db6b1fb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredPowerPointLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "ebdf2837-f566-437c-a20c-d40aba4b301f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: convert /cache3/notebook/yuyq/mooc/data/test_ppt.pptx -> /tmp/tmpgi_af5ki/test_ppt.pptx using filter : Impress MS PowerPoint 2007 XML\n"
     ]
    }
   ],
   "source": [
    "loader = UnstructuredPowerPointLoader('./data/test_ppt.pptx')\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "690bc7e1-6c5f-4ced-8c58-bf73a57544cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='本章简介\n",
      "\n",
      "大模型应用为什么需要RAG\n",
      "\n",
      "怎么学习RAG技术栈\n",
      "\n",
      "解锁RAG三大核心\n",
      "\n",
      "RAG市场前景如何\n",
      "\n",
      "大模型long context能力与RAG的争议\n",
      "\n",
      "课程实战项目介绍\n",
      "\n",
      "大模型应用遍地开花\n",
      "\n",
      "大模型应用场景：查资料、写文案、会议总结、写代码、找bug、阅读论文、数据分析\n",
      "\n",
      "国内大模型产品：文心一言、KIMI Chat、海螺AI、天工AI、秘塔搜索、360 AI搜索、秘塔写作猫\n",
      "\n",
      "国外大模型产品：GPT4-O、Gemini、Claude、GitHub Copilot\n",
      "\n",
      "大模型的两面性\n",
      "\n",
      "擅长创意创造：文案生成，故事生成，文本润色无标准答案的创意类的任务，生成式AI都表现的非常出色\n",
      "\n",
      "缺乏领域知识：受限于训练语料，甚至都不知道今天是几号\n",
      "\n",
      "id  姓名\n",
      "   1  张三\n",
      "   2  李四\n",
      "   3  王五\n",
      "   4  赵六' metadata={'source': './data/test_ppt.pptx'}\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for doc in docs:\n",
    "    print(doc)\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "b836ffc5-8846-42a5-92d6-ee6ae538e384",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pptx import Presentation\n",
    "from pptx.enum.shapes import MSO_SHAPE_TYPE\n",
    "ppt = Presentation('./data/test_ppt.pptx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "477ce375-15bb-4c86-95a2-5210558285b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slide 1:\n",
      "本章简介\n",
      "解锁RAG三大核心\n",
      "大模型应用为什么需要RAG\n",
      "大模型long context能力与RAG的争议\n",
      "RAG市场前景如何\n",
      "怎么学习RAG技术栈\n",
      "课程实战项目介绍\n",
      "Slide 2:\n",
      "国内大模型产品：文心一言、KIMI Chat、海螺AI、天工AI、秘塔搜索、360 AI搜索、秘塔写作猫\n",
      "大模型应用场景：查资料、写文案、会议总结、写代码、找bug、阅读论文、数据分析\n",
      "国外大模型产品：GPT4-O、Gemini、Claude、GitHub Copilot\n",
      "大模型应用遍地开花\n",
      "Slide 3:\n",
      "缺乏领域知识：受限于训练语料，甚至都不知道今天是几号\n",
      "擅长创意创造：文案生成，故事生成，文本润色无标准答案的创意类的任务，生成式AI都表现的非常出色\n",
      "大模型的两面性\n",
      "\n",
      "Slide 4:\n",
      "Row 1, Column 1: id\n",
      "Row 1, Column 2: 姓名\n",
      "Row 2, Column 1: 1\n",
      "Row 2, Column 2: 张三\n",
      "Row 3, Column 1: 2\n",
      "Row 3, Column 2: 李四\n",
      "Row 4, Column 1: 3\n",
      "Row 4, Column 2: 王五\n",
      "Row 5, Column 1: 4\n",
      "Row 5, Column 2: 赵六\n"
     ]
    }
   ],
   "source": [
    "for slide_number, slide in enumerate(ppt.slides, start=1):\n",
    "    print(f\"Slide {slide_number}:\")\n",
    "    for shape in slide.shapes:\n",
    "        if shape.has_text_frame:\n",
    "            print(shape.text)\n",
    "            \n",
    "        if shape.has_table:\n",
    "            table = shape.table \n",
    "            for row_idx, row in enumerate(table.rows):  \n",
    "                for col_idx, cell in enumerate(row.cells): \n",
    "                    cell_text = cell.text  \n",
    "                    print(f\"Row {row_idx + 1}, Column {col_idx + 1}: {cell_text}\")  \n",
    "            \n",
    "        if shape.shape_type == MSO_SHAPE_TYPE.PICTURE:\n",
    "                    image = shape.image\n",
    "                    image_filename = './data/test.jpg'\n",
    "                    with open(image_filename, 'wb') as f:\n",
    "                        f.write(image.blob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efa26f65-1608-4851-849d-3c5f0de41776",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "db5fa9b9-7ba3-4fd2-ae07-ea461f31e585",
   "metadata": {},
   "source": [
    "#### word\n",
    "- pip install --upgrade --quiet  docx2txt\n",
    "- pip install python-docx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "03c0d220-0630-4cca-8dfe-63840a7acce1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import Docx2txtLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "1b855def-9271-4b84-922e-552cfe2df8d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = Docx2txtLoader(\"./data/test_word.docx\")\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "b75efe4b-4632-4aef-b087-bf6743c0831a",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='大语言模型\n",
      "\n",
      "国内大模型产品：文心一言、KIMI Chat、海螺AI、天工AI、秘塔搜索、360 AI搜索、秘塔写作猫\n",
      "\n",
      "大模型应用场景：查资料、写文案、会议总结、写代码、找bug、阅读论文、数据分析\n",
      "\n",
      "国外大模型产品：GPT4-O、Gemini、Claude、GitHub Copilot\n",
      "\n",
      "大模型应用遍地开花\n",
      "\n",
      "\n",
      "\n",
      "RAG应用开发介绍\n",
      "\n",
      "本章简介\n",
      "\n",
      "解锁RAG三大核心\n",
      "\n",
      "大模型应用为什么需要RAG\n",
      "\n",
      "大模型long context能力与RAG的争议\n",
      "\n",
      "RAG市场前景如何\n",
      "\n",
      "怎么学习RAG技术栈\n",
      "\n",
      "课程实战项目介绍\n",
      "\n",
      "\n",
      "\n",
      "Id\n",
      "\n",
      "姓名\n",
      "\n",
      "1\n",
      "\n",
      "张三\n",
      "\n",
      "2\n",
      "\n",
      "李四' metadata={'source': './data/test_word.docx'}\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for doc in data:\n",
    "    print(doc)\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "b24ff445-04d9-41ef-a545-55d91957c3b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from docx import Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "c7d2fde9-9fb6-44f1-9454-80d281adcbbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "大语言模型\n",
      "国内大模型产品：文心一言、KIMI Chat、海螺AI、天工AI、秘塔搜索、360 AI搜索、秘塔写作猫\n",
      "大模型应用场景：查资料、写文案、会议总结、写代码、找bug、阅读论文、数据分析\n",
      "国外大模型产品：GPT4-O、Gemini、Claude、GitHub Copilot\n",
      "大模型应用遍地开花\n",
      "\n",
      "RAG应用开发介绍\n",
      "本章简介\n",
      "解锁RAG三大核心\n",
      "大模型应用为什么需要RAG\n",
      "大模型long context能力与RAG的争议\n",
      "RAG市场前景如何\n",
      "怎么学习RAG技术栈\n",
      "课程实战项目介绍\n",
      "\n",
      "\n",
      "\n",
      "Id | 姓名 | \n",
      "1 | 张三 | \n",
      "2 | 李四 | \n"
     ]
    }
   ],
   "source": [
    "def read_docx(file_path):\n",
    "    doc = Document(file_path)\n",
    "    for para in doc.paragraphs:\n",
    "        print(para.text)\n",
    "        \n",
    "    for table in doc.tables:\n",
    "        for row in table.rows:\n",
    "            for cell in row.cells:\n",
    "                print(cell.text, end=' | ')\n",
    "            print()\n",
    "            \n",
    "    \n",
    "# 使用示例\n",
    "file_path = './data/test_word.docx'\n",
    "read_docx(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ed79c3-0b6e-4e4e-994e-fb18f4d8a52c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### excel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eedb5c6-ed8a-47f3-b727-e342ac3b6c76",
   "metadata": {},
   "source": [
    "- pip install  openpyxl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "cda25974-5b9c-44e5-ae4b-3d702fb8cb05",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpyxl import load_workbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "eb5eeddc-baca-4004-84e8-1729d1e8da34",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/anaconda3/envs/llm/lib/python3.9/site-packages/openpyxl/worksheet/header_footer.py:48: UserWarning: Cannot parse header or footer so it will be ignored\n",
      "  warn(\"\"\"Cannot parse header or footer so it will be ignored\"\"\")\n"
     ]
    }
   ],
   "source": [
    "wb = load_workbook('./data/detail.xlsx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "b0768580-6f36-453d-a26d-7bd6fccdfa8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "附件 A1\n",
      "None B1\n",
      "None C1\n",
      "None D1\n",
      "None E1\n",
      "None F1\n",
      "None G1\n",
      "None H1\n",
      "None I1\n",
      "None J1\n",
      "None K1\n",
      "None L1\n",
      "None M1\n",
      "None N1\n",
      "None O1\n",
      "None P1\n",
      "None Q1\n"
     ]
    }
   ],
   "source": [
    "ws = wb.active\n",
    "\n",
    "for row in ws.iter_rows():\n",
    "    for cell in row:\n",
    "        print(cell.value, cell.coordinate)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "b6f25ac4-b96e-41bf-8d69-54166619266e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "B27:B28 吉林\n"
     ]
    }
   ],
   "source": [
    "for merged_range in ws.merged_cells.ranges:\n",
    "    \n",
    "    value = ws.cell(row=merged_range.min_row, column=merged_range.min_col).value\n",
    "    print(merged_range, value)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "b2e7fcb5-1724-49d5-a417-54b070e75a1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17 ['附件', None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 ['中央和国家机关工作人员赴地方差旅住宿费标准明细表', None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, None, '单位:元/人·天', None, None, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 ['序号', '地区\\n(城市)', None, '住宿费标准', None, None, '旺季地区', '旺季浮动标准', None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, None, '旺季期间', '旺季上浮价', None, None, None, None, None, None, None, None]\n",
      "17 [None, None, None, '部级', '司局级', '其他\\n人员', None, None, '部级', '司局级', '其他人员', None, None, None, None, None, None]\n",
      "17 [1, '北京', '全市', 1100, 650, 500, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [2, '天津', '6个中心城区、滨海新区、东丽区、西青区、津南区、北辰区、武清区、宝坻区、静海区、蓟县', 800, 480, 380, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '宁河区', 600, 350, 320, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [3, ' 河北', ' 石家庄市、张家口市、秦皇岛市、廊坊市、承德市、保定市', 800, 450, 350, '张家口市', '7-9月、11-3月', 1200, 675, 525, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, '秦皇岛市', '7-8月', 1200, 680, 500, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, '承德市', '7-9月', 1000, 580, 580, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 450, 310, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [4, '山西', '太原市、大同市、晋城市', 800, 480, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '临汾市', 800, 480, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '阳泉市、长治市、晋中市', 800, 480, 310, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 400, 240, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [5, None, '呼和浩特市', 800, 460, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 460, 320, ' 海拉尔市、满洲里市、阿尔山市', '7-9月', 1200, 690, 480, None, None, None, None, None, None]\n",
      "17 [None, '内蒙古', None, None, None, None, '二连浩特市', '7-9月', 1000, 580, 400, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, '额济纳旗', '9-10月', 1200, 690, 480, None, None, None, None, None, None]\n",
      "17 [6, '辽宁', '沈阳市', 800, 480, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 480, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [7, '大连', '全市', 800, 490, 350, '全市', '7-9月', 960, 590, 420, None, None, None, None, None, None]\n",
      "17 [8, '吉林', ' 长春市、吉林市、延边州、长白山管理区\\n', 800, 450, 350, '吉林市、延边州、长白山管理区', '7-9月', 960, 540, 420, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 750, 400, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [9, '黑龙江', '哈尔滨市', 800, 450, 350, '哈尔滨市', '7-9月', 960, 540, 420, None, None, None, None, None, None]\n",
      "17 [None, None, ' 其他地区', 750, 450, 300, ' 牡丹江市、伊春市、大兴安岭地区、黑河市、佳木斯市\\n', '6-8月', 900, 540, 360, None, None, None, None, None, None]\n",
      "17 [10, '上海', '全市', 1100, 600, 500, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [11, '江苏', '南京市、苏州市、无锡市、常州市、镇江市', 900, 490, 380, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 900, 490, 360, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [12, '浙江', '杭州市', 900, 500, 400, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 490, 340, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [13, '宁波', '全市', 800, 450, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [14, '安徽', '全省', 800, 460, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [15, '福建', '福州市、泉州市、平潭综合实验区', 900, 480, 380, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区\\n ', 900, 480, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [16, '厦门', '全市', 900, 500, 400, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [17, '江西', '全省', 800, 470, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [18, '山东', ' 济南市、淄博市、枣庄市、东营市、烟台\\n 市、潍坊市、济宁市、泰安市、威海市、\\n 日照市', 800, 480, 380, ' 烟台市、威海市、\\n 日照市', '7-9月', 960, 570, 450, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区\\n ', 800, 460, 360, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [19, '青岛', '全市', 800, 490, 380, '全市', '7-9月', 960, 590, 450, None, None, None, None, None, None]\n",
      "17 [20, '河南', '郑州市', 900, 480, 380, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 480, 330, '洛阳市', '4-5月上旬', 1200, 720, 500, None, None, None, None, None, None]\n",
      "17 [21, '湖北', '武汉市', 800, 480, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 480, 320, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [22, '湖南', '长沙市', 800, 450, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 450, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [23, '广东', ' 广州市、珠海市、佛山市、东莞市、中山市、江门市', 900, 550, 450, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 850, 530, 420, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [24, '深圳', '全市', 900, 550, 450, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [25, '广西', '南宁市', 800, 470, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 470, 330, ' 桂林市、北海市', '1-2月、\\n7-9月', 1040, 610, 430, None, None, None, None, None, None]\n",
      "17 [26, '海南', '海口市、三沙市、儋州市、五指山市、文昌市、琼海市、万宁市、东方市、定安县、屯昌县、澄迈县、临高县、白沙县、昌江县、乐东县、陵水县、保亭县、琼中县、洋浦开发区', 800, 500, 350, ' 海口市、文昌市、\\n 澄迈县', '11-2月', 1040, 650, 450, None, None, None, None, None, None]\n",
      "17 [None, None, None, None, None, None, ' 琼海市、万宁市、\\n 陵水县、保亭县', '11-3月', 1040, 650, 450, None, ' ', None, None, None, None]\n",
      "17 [None, None, '三亚市', 1000, 600, 400, '三亚市', '10-4月', 1200, 720, 480, None, None, None, None, None, None]\n",
      "17 [27, '重庆', ' 9个中心城区、北部新区', 800, 480, 370, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 770, 450, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [28, '四川', '成都市', 900, 470, 370, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '阿坝州、甘孜州', 800, 430, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '绵阳市、乐山市、雅安市', 800, 430, 320, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '宜宾市', 800, 430, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '凉山州', 750, 430, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '德阳市、遂宁市、巴中市', 750, 430, 310, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 750, 430, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [29, '贵州', '贵阳市', 800, 470, 370, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 750, 450, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [30, '云南', '昆明市、大理州、丽江市、迪庆州、西双版纳州', 900, 480, 380, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 900, 480, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [31, '西藏', '拉萨市', 800, 500, 350, '拉萨市', '6-9月', 1200, 750, 530, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 500, 400, 300, ' 其他地区\\n\\n ', '6-9月', 800, 500, 350, None, None, None, None, None, None]\n",
      "17 [32, '陕西', '西安市', 800, 460, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '榆林市、延安市', 680, 350, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '杨凌区', 680, 320, 260, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '咸阳市、宝鸡市', 600, 320, 260, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '渭南市、韩城市', 600, 300, 260, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 600, 300, 230, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [33, '甘肃', '兰州市', 800, 470, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 700, 450, 310, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [34, '青海', '西宁市', 800, 500, 350, '西宁市', '6-9月', 1200, 750, 530, None, None, None, None, None, None]\n",
      "17 [None, None, '玉树州、果洛州', 600, 350, 300, '玉树州', '5-9月', 900, 525, 450, None, None, None, None, None, None]\n",
      "17 [None, None, '海北州、黄南州', 600, 350, 250, ' 海北州、黄南州', '5-9月', 900, 525, 375, None, None, None, None, None, None]\n",
      "17 [None, None, '海东市、海南州', 600, 300, 250, ' 海东市、海南州', '5-9月', 900, 450, 375, None, None, None, None, None, None]\n",
      "17 [None, None, '海西州', 600, 300, 200, '海西州', '5-9月', 900, 450, 300, None, None, None, None, None, None]\n",
      "17 [35, '宁夏', '银川市', 800, 470, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '其他地区', 800, 430, 330, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [36, '新疆', '乌鲁木齐市', 800, 480, 350, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '石河子市、克拉玛依市、昌吉州、伊犁州、阿勒泰地区、博州、吐鲁番市、哈密地区、巴州、和田地区', 800, 480, 340, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '克州', 800, 480, 320, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, '喀什地区', 780, 480, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, ' 阿克苏地区', 700, 450, 300, None, None, None, None, None, None, None, None, None, None, None]\n",
      "17 [None, None, ' 塔城地区\\n ', 700, 400, 300, None, None, None, None, None, None, None, None, None, None, None]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/anaconda3/envs/llm/lib/python3.9/site-packages/openpyxl/worksheet/header_footer.py:48: UserWarning: Cannot parse header or footer so it will be ignored\n",
      "  warn(\"\"\"Cannot parse header or footer so it will be ignored\"\"\")\n"
     ]
    }
   ],
   "source": [
    "import openpyxl\n",
    "workbook = openpyxl.load_workbook('./data/detail.xlsx')\n",
    "sheet = workbook.active\n",
    "\n",
    "for row in sheet.iter_rows():\n",
    "    info = []\n",
    "    for cell in row:\n",
    "        cell_value = cell.value\n",
    "        info.append(cell_value)\n",
    "    print(len(info) ,info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f780760-b80c-46af-8303-7cf42c82650c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74a7a8f2-61a0-4110-ba94-971fd784f8ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "77540ebb-abf5-4320-b250-24e6f1be28fc",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 5.ragflow:deepdoc\n",
    "RAGflow 是开源的RAG应用，deepdoc是RAGflow的文档解析器\n",
    "\n",
    "- https://github.com/infiniflow/ragflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "ebb16699-673f-40fb-8cde-3720899494f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepdoc_parse import Pdf, chunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "ef9093aa-6a5e-4e4a-b139-7492748a4737",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None OCR is running...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Miss outlines\n",
      "INFO: Images converted.\n",
      "--- Logging error ---\n",
      "Traceback (most recent call last):\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/logging/__init__.py\", line 1083, in emit\n",
      "    msg = self.format(record)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/logging/__init__.py\", line 927, in format\n",
      "    return fmt.format(record)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/logging/__init__.py\", line 663, in format\n",
      "    record.message = record.getMessage()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/logging/__init__.py\", line 367, in getMessage\n",
      "    msg = msg % self.args\n",
      "TypeError: not all arguments converted during string formatting\n",
      "Call stack:\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/runpy.py\", line 197, in _run_module_as_main\n",
      "    return _run_code(code, main_globals, None,\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/runpy.py\", line 87, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel_launcher.py\", line 17, in <module>\n",
      "    app.launch_new_instance()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/traitlets/config/application.py\", line 992, in launch_instance\n",
      "    app.start()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/kernelapp.py\", line 701, in start\n",
      "    self.io_loop.start()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/tornado/platform/asyncio.py\", line 195, in start\n",
      "    self.asyncio_loop.run_forever()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/asyncio/base_events.py\", line 601, in run_forever\n",
      "    self._run_once()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/nest_asyncio.py\", line 133, in _run_once\n",
      "    handle._run()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/asyncio/events.py\", line 80, in _run\n",
      "    self._context.run(self._callback, *self._args)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 534, in dispatch_queue\n",
      "    await self.process_one()\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 523, in process_one\n",
      "    await dispatch(*args)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 429, in dispatch_shell\n",
      "    await result\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 767, in execute_request\n",
      "    reply_content = await reply_content\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/ipkernel.py\", line 429, in do_execute\n",
      "    res = shell.run_cell(\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell\n",
      "    return super().run_cell(*args, **kwargs)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3024, in run_cell\n",
      "    result = self._run_cell(\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3079, in _run_cell\n",
      "    result = runner(coro)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/async_helpers.py\", line 129, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3284, in run_cell_async\n",
      "    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3466, in run_ast_nodes\n",
      "    if await self.run_code(code, result, async_=asy):\n",
      "  File \"/root/anaconda3/envs/llm/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3526, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"/tmp/ipykernel_17407/1685824862.py\", line 4, in <module>\n",
      "    res = chunk('./data/zhidu_travel.pdf', callback=dummy)\n",
      "  File \"/cache3/notebook/yuyq/mooc/deepdoc_parse.py\", line 69, in chunk\n",
      "    sections, tbls = pdf_parser(filename if not binary else binary,\n",
      "  File \"/cache3/notebook/yuyq/mooc/deepdoc_parse.py\", line 24, in __call__\n",
      "    self.__images__(\n",
      "  File \"/cache3/notebook/yuyq/mooc/deepdoc/parser/pdf_parser.py\", line 1020, in __images__\n",
      "    logging.info(\"Is it English:\", self.is_english)\n",
      "Message: 'Is it English:'\n",
      "Arguments: (False,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None OCR finished.\n",
      "OCR: 0.871143713593483\n",
      "preprocess\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: Table processing...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.65 Layout analysis finished.\n",
      "layouts: 1.1523799449205399\n",
      "preprocess\n",
      "0.67 Table analysis finished.\n",
      "0.68 Text merging finished\n"
     ]
    }
   ],
   "source": [
    "def dummy(prog=None, msg=\"\"):\n",
    "        print(prog, msg)\n",
    "\n",
    "res = chunk('./data/zhidu_travel.pdf', callback=dummy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "e8d618b9-e10c-4f8e-a3f7-07aa6098aaa3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========\n",
      "<table>\n",
      "<tr><td  >级别</td><td  >火车 （含高铁、动车、全列软席列车）</td><td  >轮船 （不包括 旅游船）</td><td  >飞机</td><td  >其他交通工具 （不包括出租 小汽车）</td></tr>\n",
      "<tr><td  >享受副部级 待遇及以上 人员</td><td  >火车软席（软座、软卧），高铁/动车商 务座，全列软席列车一等软座</td><td  >一等舱</td><td  >头等舱</td><td  >凭据报销</td></tr>\n",
      "<tr><td  >秘书长及副 秘书长</td><td  >火车软席（软座、软卧），高铁/动车一 等座，全列软席列车一等软座</td><td  >二等舱</td><td  >经济舱</td><td  >凭据报销</td></tr>\n",
      "<tr><td  >其余人员</td><td  >火车硬席（硬座、硬卧），高铁/动车二 等座、全列软席列车二等软座</td><td  >三等舱</td><td  >经济舱</td><td  >凭据报销</td></tr>\n",
      "</table>\n",
      "==========\n",
      "<table><caption>伙食补助费参考以下标准：</caption>\n",
      "<tr><td  >地区</td><td  >伙食补助费标准</td></tr>\n",
      "<tr><td  >西藏、青海、新疆</td><td  >120 元/人、天</td></tr>\n",
      "<tr><td  >其他省份</td><td  >100 元/人、天</td></tr>\n",
      "</table>\n",
      "==========\n",
      "差旅费用标准\n",
      "差旅费开支范围包括工作人员临时到常驻地以外地区公务出差所发生的城市间交通费、住宿费、伙食补助费和市内交通费。一、城市间交通费城市间交通费是指工作人员因公到常驻地以外地区出差乘坐火车、轮船、飞机等交通工具所发生的费用。1.出差人员在不影响公务、确保安全的前提下，选乘经济便捷的交通工具。2.出差人员要按照规定等级乘坐交通工具，未按规定乘坐的，超支部分自理。乘坐交通工具的等级见下表：\n",
      "<table>\n",
      "<tr><td  >级别</td><td  >火车 （含高铁、动车、全列软席列车）</td><td  >轮船 （不包括 旅游船）</td><td  >飞机</td><td  >其他交通工具 （不包括出租 小汽车）</td></tr>\n",
      "<tr><td  >享受副部级 待遇及以上 人员</td><td  >火车软席（软座、软卧），高铁/动车商 务座，全列软席列车一等软座</td><td  >一等舱</td><td  >头等舱</td><td  >凭据报销</td></tr>\n",
      "<tr><td  >秘书长及副 秘书长</td><td  >火车软席（软座、软卧），高铁/动车一 等座，全列软席列车一等软座</td><td  >二等舱</td><td  >经济舱</td><td  >凭据报销</td></tr>\n",
      "<tr><td  >其余人员</td><td  >火车硬席（硬座、硬卧），高铁/动车二 等座、全列软席列车二等软座</td><td  >三等舱</td><td  >经济舱</td><td  >凭据报销</td></tr>\n",
      "</table>\n",
      "备注：\n",
      "享受副部级待遇及以上人员出差，因工作需要，随行一人可乘坐同等级交通工具；乘坐飞机的，民航发展基金、燃油附加费可以凭据报销；乘坐飞机、火车、轮船等交通工具的，每人次可以购买交通意外保险一份。由我会统一购买交通意外保险的，不再重复购买。3.我会工作人员出差，高铁最短5 个小时内能够到达目的地的，原则上应乘坐高铁，如遇特殊情况，经分管副秘书长、秘书长审批后，可乘坐飞机。4.国内出差购买机票,应当按照厉行节约和支持本国航空公司发展的原则,优先购买通过政府采购方式确定的我国航空公司航班优惠机票。飞机票应通过定点供应商采购。二、住宿费住宿费是指工作人员因公出差期间入住宾馆（包括饭店、招待所，下同）发生的房租费用。享受副部级待遇及以上人员住普通套间，秘书长及以下人员住单间或标准间。出差人员应当在职务级别对应的住宿费标准限额内，选择安全、经济、便捷的宾馆住宿。国内出差发生的住宿，原则上应当按照住宿费标准执行，如遇特殊情况（如对方组织单位指定或推荐酒店等情况），超出费用标准的，须事先报分管资金的副秘书长、秘书长批准后方可报销。住宿费按照财政部《中央和国家机关工作人员赴地方差旅住宿费标准明细表》（财行[2016]71号）执行。三、伙食补助费伙食补助费是指对工作人员在因公出差期间给予的伙食补助费用。伙食补助费按出差自然(日历)天数计算，按规定标准包干使用。出差人员应当自行用餐。凡由接待单位统一安排用餐的，不再发放伙食补助费。\t\n",
      "==========\n",
      "2\n",
      "<table><caption>伙食补助费参考以下标准：</caption>\n",
      "<tr><td  >地区</td><td  >伙食补助费标准</td></tr>\n",
      "<tr><td  >西藏、青海、新疆</td><td  >120 元/人、天</td></tr>\n",
      "<tr><td  >其他省份</td><td  >100 元/人、天</td></tr>\n",
      "</table>\n",
      "==========\n",
      "四、市内交通费\n",
      "市内交通费是指工作人员因公出差期间发生的市内交通费用。出差人员的市内交通费按出差的自然（日历）天数计算，每人每天80 元包干使用。由接待单位或其他单位提供交通工具的，不再发放市内交通费。\n"
     ]
    }
   ],
   "source": [
    "for data in res:\n",
    "    print(\"=\"*10)\n",
    "    print(data['content_with_weight'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abec4206-5126-45cb-ade6-9bac37798c92",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "924db9d7-2585-4211-a080-33955c8657c1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5f5b038d-f6eb-4aad-b32c-94b35b5f4152",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 6.textsplit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "5f2b5ad2-d8e0-4662-9939-40a002d8fe38",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from langchain_community.document_loaders import PyMuPDFLoader\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "from langchain_experimental.text_splitter import SemanticChunker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "87c8399c-fabc-4256-ae5e-74cd2ee90ce7",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_path = './data/zhidu_travel.pdf'\n",
    "loader = PyMuPDFLoader(doc_path)\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "2195cd57-f020-4487-8361-74e5ec31331b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "for doc in docs:\n",
    "    data += doc.page_content.split('\\u3002')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "aa223388-ff22-424d-8c39-4fb85934a8ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_nums = [len(txt) for txt in data if len(txt) != 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "8798e4cc-6525-4f97-bdd7-19d6e0f5d30f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51.166666666666664"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(word_nums)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "65445aa1-6318-4311-82c0-02f4c3c2978a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.min(word_nums)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "51952b25-3840-4796-b622-15851c4c2ad4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "242"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(word_nums)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "ae5ddddf-6aa7-411e-b719-df6cc936fbf3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100.79999999999998"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(word_nums, q=95)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d54579-11ff-40d0-8cf7-c82781a6d011",
   "metadata": {},
   "source": [
    "#### 递归文本分块：RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "0ac74f1d-64f2-46e5-b4b7-9880737b18c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "r_spliter = RecursiveCharacterTextSplitter(chunk_size=128,\n",
    "                                          chunk_overlap=30,\n",
    "                                          separators=[\"\\n\\n\",\n",
    "                                                      \"\\n\", \n",
    "                                                      \".\",\n",
    "                                                      \"\\uff0e\",  # Fullwidth full stop\n",
    "                                                       \"\\u3002\",  # Ideographic full stop\n",
    "                                                      \",\",\n",
    "                                                      \"\\uff0c\",  # Fullwidth comma\n",
    "                                                      \"\\u3001\",  # Ideographic comma\n",
    "                                                     ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "3fd973ae-798f-4f85-8856-9763366a41b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "split_docs = loader.load_and_split(r_spliter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "abc8720b-fb60-40d6-8da5-9b934e42c4bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "差旅费用标准\n",
      "差旅费开支范围包括工作人员临时到常驻地以外地区\n",
      "公务出差所发生的城市间交通费、住宿费、伙食补助费和市\n",
      "内交通费。\n",
      "一、城市间交通费\n",
      "城市间交通费是指工作人员因公到常驻地以外地区出\n",
      "差乘坐火车、轮船、飞机等交通工具所发生的费用。\n"
     ]
    }
   ],
   "source": [
    "print(split_docs[0].page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "b817532c-3201-4a31-b993-3253bc74e3ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "差乘坐火车、轮船、飞机等交通工具所发生的费用。\n",
      "1.出差人员在不影响公务、确保安全的前提下，选乘经\n",
      "济便捷的交通工具。\n",
      "2.出差人员要按照规定等级乘坐交通工具，未按规定乘\n",
      "坐的，超支部分自理。乘坐交通工具的等级见下表：\n",
      "级别\n",
      "火车\n"
     ]
    }
   ],
   "source": [
    "print(split_docs[1].page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "7f6e9fda-d3ab-4ccb-8b56-74c6568b11d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "4e37ba80-1644-4669-923d-068a9557c857",
   "metadata": {},
   "outputs": [],
   "source": [
    "split_docs = r_spliter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "30d82853-52d7-4eb5-8e29-41d195769d1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "差旅费用标准\n",
      "差旅费开支范围包括工作人员临时到常驻地以外地区\n",
      "公务出差所发生的城市间交通费、住宿费、伙食补助费和市\n",
      "内交通费。\n",
      "一、城市间交通费\n",
      "城市间交通费是指工作人员因公到常驻地以外地区出\n",
      "差乘坐火车、轮船、飞机等交通工具所发生的费用。\n"
     ]
    }
   ],
   "source": [
    "print(split_docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86448ca5-ae30-4587-86df-72b092ab7192",
   "metadata": {},
   "source": [
    "#### 基于向量的语义分块：SemanticChunker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "7eb67b8d-8f60-46b2-bcca-0cbd2956136b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_17407/339235600.py:3: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 1.0. An updated version of the class exists in the langchain-huggingface package and should be used instead. To use it run `pip install -U langchain-huggingface` and import as `from langchain_huggingface import HuggingFaceEmbeddings`.\n",
      "  embeddings = HuggingFaceEmbeddings(model_name=model_path,\n",
      "INFO: PyTorch version 2.0.1 available.\n",
      "INFO: Load pretrained SentenceTransformer: ./data/llm_app/embedding_models/bge-m3/\n",
      "/root/anaconda3/envs/llm/lib/python3.9/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
      "  return self.fget.__get__(instance, owner)()\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "model_path='./data/llm_app/embedding_models/bge-m3/'\n",
    "embeddings = HuggingFaceEmbeddings(model_name=model_path,\n",
    "                                   model_kwargs={'device': \"cpu\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "a216cda3-6975-4cc7-948f-ccf7f396894a",
   "metadata": {},
   "outputs": [],
   "source": [
    "semantic_chunker = SemanticChunker(embeddings, \n",
    "                                   breakpoint_threshold_type=\"percentile\",\n",
    "                                   sentence_split_regex='(?<=[。；：])',\n",
    "                                   buffer_size=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "c47a6e56-5657-4882-8494-b921b7db4d1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "semantic_docs = semantic_chunker.create_documents([docs[0].page_content])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "6a470495-c601-4bd7-931a-0aa4a34098dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "差旅费用标准\n",
      "差旅费开支范围包括工作人员临时到常驻地以外地区\n",
      "公务出差所发生的城市间交通费、住宿费、伙食补助费和市\n",
      "内交通费。 \n",
      "一、城市间交通费\n",
      "城市间交通费是指工作人员因公到常驻地以外地区出\n",
      "差乘坐火车、轮船、飞机等交通工具所发生的费用。 \n",
      "1.出差人员在不影响公务、确保安全的前提下，选乘经\n",
      "济便捷的交通工具。 \n",
      "2.出差人员要按照规定等级乘坐交通工具，未按规定乘\n",
      "坐的，超支部分自理。\n",
      "198\n",
      "====================================================================================================\n",
      "乘坐交通工具的等级见下表： \n",
      "级别\n",
      "火车\n",
      "（含高铁、动车、全列软席列车）\n",
      "轮船\n",
      "（不包括\n",
      "旅游船）\n",
      "飞机\n",
      "其他交通工具\n",
      "（不包括出租\n",
      "小汽车）\n",
      "享受副部级\n",
      "待遇及以上\n",
      "人员\n",
      "火车软席（软座、软卧）\n",
      "，高铁/动车商\n",
      "务座，全列软席列车一等软座\n",
      "一等舱\n",
      "头等舱\n",
      "凭据报销\n",
      "秘书长及副\n",
      "秘书长\n",
      "火车软席（软座、软卧）\n",
      "，高铁/动车一\n",
      "等座，全列软席列车一等软座\n",
      "二等舱\n",
      "经济舱\n",
      "凭据报销\n",
      "其余人员\n",
      "火车硬席（硬座、硬卧）\n",
      "，高铁/动车二\n",
      "等座、全列软席列车二等软座\n",
      "三等舱\n",
      "经济舱\n",
      "凭据报销\n",
      "\n",
      "243\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "for semantic_chunk in semantic_docs:\n",
    "    print(semantic_chunk.page_content)\n",
    "    print(len(semantic_chunk.page_content))\n",
    "    print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72944b04-b0a9-43ba-8952-706b20482ede",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f55e65-397a-4829-bc10-e3632c37781e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f26dde2-4322-4846-abaa-d24d1586e51f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd175d2f-9c1a-4598-b20a-94ab4fb3722b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d8374804-0e16-4648-b1fa-144d007d0a1b",
   "metadata": {},
   "source": [
    "#### 基于模型的语义分块：nlp_bert_document-segmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "0373adcc-2fc8-40a8-8954-d24a86442dc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    }
   ],
   "source": [
    "from modelscope.outputs import OutputKeys\n",
    "from modelscope.pipelines import pipeline\n",
    "from modelscope.utils.constant import Tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06a8582f-ba8e-4e9b-82c1-d190ddb40688",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "6018cd22-b4e6-467c-9d78-9654351dfab2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-10-06 16:02:31,626 - modelscope - WARNING - Model revision not specified, use revision: v1.0.1\n",
      "2024-10-06 16:02:31,974 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/nlp_bert_document-segmentation_chinese-base\n",
      "2024-10-06 16:02:31,975 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/nlp_bert_document-segmentation_chinese-base.\n",
      "2024-10-06 16:02:31,976 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/nlp_bert_document-segmentation_chinese-base\n",
      "/root/anaconda3/envs/llm/lib/python3.9/site-packages/transformers/configuration_utils.py:364: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.\n",
      "  warnings.warn(\n",
      "2024-10-06 16:02:34,434 - modelscope - WARNING - No preprocessor field found in cfg.\n",
      "2024-10-06 16:02:34,436 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.\n",
      "2024-10-06 16:02:34,436 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/root/.cache/modelscope/hub/damo/nlp_bert_document-segmentation_chinese-base'}. trying to build by task and model information.\n",
      "2024-10-06 16:02:34,437 - modelscope - WARNING - No preprocessor key ('bert-for-document-segmentation', 'document-segmentation') found in PREPROCESSOR_MAP, skip building preprocessor.\n"
     ]
    }
   ],
   "source": [
    "p = pipeline(\n",
    "    task=Tasks.document_segmentation,\n",
    "    model='damo/nlp_bert_document-segmentation_chinese-base')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "61b6c6e6-ccc1-4b50-a63e-fa45b93745cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/anaconda3/envs/llm/lib/python3.9/site-packages/transformers/modeling_utils.py:1052: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "result = p(documents=docs[1].page_content.replace('\\n', ''))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "e0cbf2a6-06e5-46ec-9fe0-6ee2c729d7ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t2备注：享受副部级待遇及以上人员出差，因工作需要，随行一人可乘坐同等级交通工具；乘坐飞机的，民航发展基金、燃油附加费可以凭据报销；乘坐飞机、火车、轮船等交通工具的，每人次可以购买交通意外保险一份。由我会统一购买交通意外保险的，不再重复购买。\n",
      "\t3.\n",
      "\t我会工作人员出差，高铁最短5 个小时内能够到达目的地的，原则上应乘坐高铁，如遇特殊情况，经分管副秘书长、秘书长审批后，可乘坐飞机。\n",
      "\t4.\n",
      "\t国内出差购买机票,应当按照厉行节约和支持本国航空公司发展的原则,优先购买通过政府采购方式确定的我国航空公司航班优惠机票。飞机票应通过定点供应商采购。\n",
      "\t二、住宿费住宿费是指工作人员因公出差期间入住宾馆（包括饭店、招待所，下同）发生的房租费用。享受副部级待遇及以上人员住普通套间，秘书长及以下人员住单间或标准间。出差人员应当在职务级别对应的住宿费标准限额内，选择安全、经济、便捷的宾馆住宿。国内出差发生的住宿，原则上应当按照住宿费标准执行，如遇特殊情况（如对方组织单位指定或推荐酒店等情况），超出费用标准的，须事先报分管资金的副秘书长、秘书长批准后方可报销。住宿费按照财政部《中央和国家机关工作人员赴地方差旅住宿费标准明细表》（财行[2016]71 号）执行。\n",
      "\t\n"
     ]
    }
   ],
   "source": [
    "print(result['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e226d621-be02-4792-b291-e01a245adb35",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm",
   "language": "python",
   "name": "llm"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
